Skip to content

fix(ci): recover hosted macOS Docker runtime lost mid-scenario - #252

Merged
jsugg merged 1 commit into
mainfrom
fix/ci-macos-runtime-loss-recovery
Nov 11, 2026
Merged

jsugg merged 1 commit into
mainfrom
fix/ci-macos-runtime-loss-recovery

Conversation

@jsugg

@jsugg jsugg commented Nov 11, 2026 •

Copy link
Copy Markdown
Owner

Summary

Hosted macOS fresh-host scenarios now survive the OrbStack Docker backend dying mid-run: a failed Docker-backed step is probed, the runtime phase's OrbStack recovery runs inside the scenario's job budget, and the step is retried once. A CI-only fault drill proves it on a real runner.

Root cause: what failed in main run 42971397902

Job "macOS Fresh Host Sidecars (openclaw-default)", step "Run macOS fresh-host scenario":

Time (UTC) Event (source)
06:40:28 sidecars up starts its second compose call, the LiteLLM schema bootstrap (docker compose run --rm litellm --skip_server_startup) (job log)
06:40:35.305 OrbStack Helper cancels its own com.apple.Virtualization.VirtualMachine XPC connection (0x600001ca81e0, opened 06:21:21) (orbstack-unified.log)
06:40:35.506 error during connect: Get ".../docker.sock/_ping": EOF (job log)
06:40:35.756 launchd: service inactive: com.apple.Virtualization.VirtualMachine.7D08… (orbstack-unified.log)
06:40:36.67 OrbStack Helper runs its exit handler (orbstack-unified.log)
06:40:46–55 retry's sidecars down gets CommandError: Docker is installed but the backend is not reachable, then 5 backend probes fail and the job fails (job log)

Post-mortem state: orb status = Stopped, socket file gone, docker info fails with "no such file or directory".

What the evidence shows: OrbStack shut down its own VM. The VM manager cancelled its Virtualization connection before the socket EOF and before launchd marked the VM inactive. That is a VM-manager-initiated stop, not the host killing the VM process from outside.

Ruled out:

  • Host memory pressure. vm_stat counters are cumulative since boot: Swapouts: 0, Swapins: 0, 97 compressions in total, ~4.7 GiB free. The host never swapped at any point in the job, including at 06:40:35.
  • Disk. 102 GiB available; the workflow cache was 447 MiB.

Not determinable from this run: why the guest or VM manager stopped (guest kernel panic, guest OOM of an OrbStack agent, or a VM manager fault). The unified-log predicate only matched host-side OrbStack processes, and OrbStack's own ~/.orbstack/log/vmgr.log and gui.log were never collected. So no memory-limit or disk change is made: nothing in the evidence supports one. Instead, diagnostics now collect that evidence next time. This was 1 failure in the last 60 CI runs; the same code passed on #251 and 7a4eb20.

Design

  • Classification by live probe, not by error text. recover_lost_runtime (in _hosted_docker/runtime.py) reuses fix(ci): bound hosted docker waits by job-budget deadlines #238's _recover_orbstack_runtime. A probe extracted from _classify_runtime_failure decides whether the backend is gone. A step failing with, say, "timed out waiting for compose service" while Docker answers is passed through untouched.
  • Budget. scenario_phase_deadline = min(scenario start + scenario reserve, job deadline − observer reserve), which equals the scenario step's own timeout. Recovery gets a slice of it, capped at the runtime phase's 360s recovery slice and always leaving SCENARIO_RETRY_RESERVE_SECONDS = 600s for the retried step and later phases. The slowest observed exercise-sidecars was 411s, plus 41s of later phases.
  • Guard (_fresh_host/runtime_loss.py). It wraps exercise-sidecars, exercise-channels-runtime and exercise-browser-lab. On recovery it retries once. The channels retry restarts the sidecars stack, because the lost VM took it down. It is armed only for macOS + OrbStack + a recorded job budget. Linux, local and budget-less runs keep their current fail-fast behavior. Product code in src/clawops is unchanged and still raises its clear error.
  • Evidence. diagnostics/runtime-loss/<step>/{report.json,attempt-NN/{before,after},recovery-command.txt}. Diagnostics also now copy the tail of each ~/.orbstack/log/*.log and widen the unified-log predicate to com.apple.Virtualization* and kernel memorystatus kills.
  • Fault drill. FRESH_HOST_FAULT_INJECTION=orbstack-stop-during-sidecars runs orb stop once, right before the sidecars stack starts. A marker in diagnostics makes it one-shot and timestamped. It is gated three ways:
    • it is an input on the Acceptance Tests workflow_dispatch only, not on workflow_call;
    • the core step's env expression requires workflow_dispatch and a ref other than refs/heads/main;
    • the helper re-checks GITHUB_EVENT_NAME/GITHUB_REF and ignores the request otherwise.

Fault-injection proof

Run: https://github.com/jsugg/strongclaw/actions/runs/42982265106. It was dispatched on this branch with fault_injection=orbstack-stop-during-sidecars and is green (attempt 2).

Attempt 1's hypermemory Sidecars job hit "The hosted runner lost communication with the server" during actions/setup-node, about 15s into the job and before any helper code ran. Only that job was rerun, with the same inputs.

macOS Fresh Host Sidecars (openclaw-default), attempt 1. The first lines reproduce the original failure exactly; the run then recovers:

08:06:18.097 [fresh-host] Fault injection orbstack-stop-during-sidecars: stopping OrbStack.
08:06:18.097 [fresh-host] Running: orb stop
08:06:29.475 [fresh-host] Command exited with 1 (attempt 1/2); reclaiming resources and retrying: ... sidecars up --repo-local-state
08:06:29.721 clawops.strongclaw_runtime.CommandError: Docker is installed but the backend is not reachable from this shell. ...
08:06:39.464 [fresh-host] Docker backend not ready yet (attempt 4/5); retrying.
08:06:41.680 [fresh-host] exercise-sidecars failed (Docker backend is not reachable from this shell.); probing the Docker runtime for loss.
08:06:41.682 [hosted-docker] Docker runtime lost during exercise-sidecars: orbstack_socket_missing.
08:06:41.683 [hosted-docker] OrbStack recovery attempt 1/2 triggered by orbstack_socket_missing; 180s slice.
08:06:59.564 [hosted-docker] Running: orb start
08:07:32.613 [hosted-docker] Docker ready after 1 probe(s).
08:07:47.292 [fresh-host] Docker runtime recovered; retrying exercise-sidecars once.
08:13:43.734 [fresh-host] Starting phase=exercise-channels-runtime.
08:13:54.477 [fresh-host] Starting phase=exercise-recovery-smoke.

macOS Fresh Host Sidecars (hypermemory), attempt 2:

09:01:57.754 [fresh-host] Fault injection orbstack-stop-during-sidecars: stopping OrbStack.
09:02:21.473 [hosted-docker] Docker runtime lost during exercise-sidecars: orbstack_socket_missing.
09:02:31.090 [hosted-docker] Running: orb start
09:03:11.860 [fresh-host] Docker runtime recovered; retrying exercise-sidecars once.
09:11:58.249 [fresh-host] Starting phase=exercise-channels-runtime.

Artifacts contain:

  • diagnostics/runtime-loss/exercise-sidecars/report.json: recovered: true, trigger_reason: orbstack_socket_missing, deadline_limited_by: runtime-loss-cap, commands [["orb","start"]].
  • attempt-01/{before,after} diagnostics.
  • the fault-injection/orbstack-stop-during-sidecars.json marker.
  • orbstack-logs/vmgr*.log. The rotated vmgr.1.log records the stop as Wait for services to exit → reboot: Power down → vmgr [VM] stopped, the evidence that was missing for run 42971397902.

Normal PR CI (switch off): https://github.com/jsugg/strongclaw/actions/runs/42982302893, all green. The sidecars job shows FRESH_HOST_FAULT_INJECTION: (empty) and Runtime-loss recovery armed: 2100s scenario window, with no fault and no loss.

Tests

  • tests/suites/unit/ci/test_fresh_host_runtime_loss.py (fake clock): arming (macOS+budget, no budget, Linux), observer-reserve disarm, pass-through without a window, Linux pass-through with a budget, a reachable backend is not runtime loss, socket-EOF recovery and a single retry, missing socket uses orb start, retry action, refusal when only the retry reserve is left, exhaustion mid-recovery, failed recovery with attempt evidence, unrecoverable reason, and an end-to-end sidecars phase with an injected orb stop.
  • tests/suites/unit/ci/test_fresh_host_faults.py: event/ref gating, off values, unknown fault, one-shot firing, inert on push to main.
  • test_fresh_host_budget.py: scenario deadline cap, late start, exhaustion.
  • test_hosted_docker.py: OrbStack log tails and the widened predicate.
  • Contract test_fault_injection_cannot_be_enabled_on_push_to_main.

Local gates (raw): make lint (ruff: all checks passed), make typecheck (mypy: no issues in 378 files; pyright: 0 errors), make imports, make test-unit (1220 passed), make test-contracts (150 passed), actionlint clean.

The runtime and image phases already recover OrbStack, but nothing did
once the scenario started. In main run 42971397902 the OrbStack VM
stopped seven seconds into the LiteLLM schema bootstrap of
`sidecars up`, and the job failed after five 2s backend probes.

Scenario steps that drive Docker (exercise-sidecars,
exercise-channels-runtime, exercise-browser-lab) now run through a
runtime-loss guard on hosted macOS OrbStack runs with a recorded job
budget. On failure a live probe decides whether the backend is gone;
if so the runtime phase's OrbStack recovery runs inside
min(scenario start + scenario reserve, job deadline - observer
reserve), capped at the runtime recovery slice and leaving 600s for
the retried step, then the step is retried once. Linux, local and
budget-less runs keep failing fast; product code is unchanged.

Diagnostics now keep the tail of OrbStack's own logs and widen the
unified-log predicate to the Virtualization framework and kernel
memorystatus kills, the evidence the failed run lacked.

A CI-only drill, FRESH_HOST_FAULT_INJECTION=orbstack-stop-during-sidecars,
runs `orb stop` before the sidecars stack starts. It is reachable only
from a manual Acceptance Tests dispatch off main and is re-checked by
the helper.
@jsugg
jsugg merged commit 548c3b4 into main Nov 11, 2026
33 of 34 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant